Papers with RoBERTa model
Optimizing Entity Resolution in Voice Interfaces: An ASR-Aware Entity Reference Expansion Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) errors in voice-based dialog systems pose significant impediments to downstream tasks. |
| Approach: | They propose an automatic speech recognition (ASR) error-aware loss function to inject failed mentions and resolved entity names into the knowledge graph to enhance its awareness of unresolved mentions. |
| Outcome: | The proposed system enhances the knowledge graph's awareness of unresolved mentions by injecting pairs of failed mentions and resolved entities into the knowledge map. |
Shuffled-token Detection for Refining Pre-trained RoBERTa (2021.naacl-srw)
Copied to clipboard
| Challenge: | State-of-the-art transformer models have achieved robust performance on a variety of NLP tasks. |
| Approach: | They propose to refine a pre-trained NLP model by detecting shuffled tokens . they use a sequential approach to train a model using random shuffling . |
| Outcome: | The proposed model achieves better performance on 4 out of 7 GLUE tasks. |
Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Large-scale conversational assistants process every utterance using multiple models for domain, intent and named entity recognition. |
| Approach: | They combine utterance encodings from a RoBERTa model with the Nbest hypothesis produced by the production system to detect domain classification errors. |
| Outcome: | The proposed approach outperforms bi-LSTM models and a standalone model by 2.2% to 32.2% by ensembling multiple models. |
Challenging America: Modeling language in longer time scales (2022.findings-naacl)
Copied to clipboard
Jakub Pokrywka, Filip Graliński, Krzysztof Jassem, Karol Kaczmarek, Krzysztof Jurkiewicz, Piotr Wierzchon
| Challenge: | a dominant approach to solving NLP tasks is pre-training a large neural language model and fine-tuning the model for specific tasks. |
| Approach: | They propose a challenge to train and fine-tune large Transformer models for historical texts . they pre-trained a RoBERTa model from scratch from the historical texts and evaluate them on benchmarks . |
| Outcome: | The proposed ML task is based on OCR-ed clippings from the Chronicling America portal. |
Enhancing Job Evaluation with Data Augmentation and Text Classification (2026.acl-industry)
Copied to clipboard
| Challenge: | Recruiters rely on job titles, role descriptions, and responsibility levels to determine job grades and salary structures. |
| Approach: | They propose to semi-automate job evaluation by fine-tuning a RoBERTa model for classification and using Gemini to generate synthetic job descriptions for rare job titles. |
| Outcome: | The proposed method improves job evaluation by boosting consistency and speeding up workflows. |
Zero-shot Fact Verification by Claim Generation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods for fact verification require large datasets, which can be expensive. |
| Approach: | They propose a framework for training a robust fact verification model by using automatically generated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. |
| Outcome: | The proposed framework reduces the demand for human-annotated training data and improves a model's F1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples. |
Constructions are Revealed in Word Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Construction grammar posits that constructions are form-meaning pairings that are acquired through experience with language. |
| Approach: | They propose to use a RoBERTa model to model constructions as patterns of statistical affinity . they show that statistical affinity is likely an important, but partial, signal available to learners . |
| Outcome: | The proposed model shows that constructions will be revealed as patterns of statistical affinity . the proposed model is based on a model that is able to distinguish constructions from text . |
Can Transformer Models Measure Coherence In Text: Re-Thinking the Shuffle Test (2021.acl-short)
Copied to clipboard
| Challenge: | Recent work shows that modern NLP models can detect shuffled text without supervision. |
| Approach: | They propose to use Shuffle Test to evaluate whether NLP models can measure coherence in text . they argue that this is unlikely to lead to a good model of text coherency . |
| Outcome: | The Shuffle Test is the most common task to evaluate whether NLP models can measure coherence in text. |
Partial-input baselines show that NLI models can ignore context, but they don’t. (2022.naacl-main)
Copied to clipboard
| Challenge: | Researchers have shown that many datasets contain statistical biases, or "annotation artifacts" that systems leverage to correctly predict entailment. |
| Approach: | They propose to use edited contexts to examine RoBERTa models' sensitivity to edited context to examine their model's sensitivity. |
| Outcome: | The proposed model can learn to condition on context, despite being trained on artifact-ridden datasets. |
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be fine-tuned to produce state-of-the-art results for a wide range of language understanding tasks. |
| Approach: | They propose to analyze fine-tuning through the lens of intrinsic dimension . they show that pre-trained models have a low intrinsic dimension reparameterization . |
| Outcome: | The proposed model can achieve 90% of the full parameter performance levels on MRPC with low data regime. |
A Simple Yet Strong Pipeline for HotpotQA (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-hop question answering have been proposed with varying complexities. |
| Approach: | They propose to use BERT to identify potentially relevant sentences independently of each other . they feed selected sentences into a standard BERT span prediction model to choose an answer . |
| Outcome: | The proposed pipeline outperforms existing models on hotpotQA and support identification. |